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ABSTRACT 

Due to the rapid growth in Internet resources, mobile technologies and social media, teaching and learning are 
increasingly adapting to the notion that ’content is open; learners are social'. The learning materials are open but 
effective learning is challenging due to the explosion of unstructured content on the web. The effectiveness of 
learning on the web largely depends on the relevancy of the content and the learner's engagement. This paper's 
objective is to develop an Open Content Social Learning(OCSL) system, to compare different pedagogical 
strategies and algorithms on improving effective learning. This paper proposes an enhanced learner-centered 
online learning experience by matching the content based on learning goals, historical learning preferences and 
behaviors from other learners with similar goals to increase the learner interaction and engagement. 

INTRODUCTION 

Open Educational Resources (OERs) are teaching and learning materials that anyone can use and share freely, 
without charge. Since first being coined by UNESCO in 2002, the term Open Educational Resources has evolved 
to meet the fast pace of the change and the diverse contexts in which it has now been used (Bossu, Bull, & 
Brown, 2012). The worldwide OER movement is rooted in the idea of high quality education at no cost. The 
Cape Town Declaration (2007) states that “Educators worldwide are developing a vast pool of educational 
resources on the Internet, open and free for all to use. These educators are creating a world where each and every 
person on earth can access and contribute to the sum of all human knowledge. They are also planting the seeds of 
a new pedagogy where educators and learners create, shape and evolve knowledge together, deepening their 
skills and understanding as they go.” 

Open learning enables learners to be self-determined and interest-guided. Stacey (2013) educators to “Go beyond 
open enrollments and use open pedagogies that leverage the entire web not just the specific content in the 
MOOC platform”. Learners are often unable identify which material is needed, useful, and required at their 
level. Hence, open content learning design must assimilate the material from various sources and provide a new 
pedagogy that is appropriate to the needs of today’s learners (Smyth, Bossu & Stagg, 2015). This paper explains 
the design for an Open Content Social Learning (OCSL) system that leverages Open Content to deliver an 
adaptive and personalized experience accounting for the pedagogical needs of the learners and similar learners 
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and the need to recommend learning activities in a pedagogically effective order. 

RELATED RESEARCH 

Learner’s experiences with open learning do not always contribute to effective learning because some traditional 
pedagogical strategies are still being used. Over the past decade, researchers have investigated different 
pedagogical strategies for making the online learning environment effective. Sathiyamurthy & Geetha (2012) 
state that “The effectiveness of an e-leaming system for distance education to a large extent depends on the 
relevancy and presentation of learning content to the learner”. In a recent study, Kim & Reeves (2007) showed 
that the increase in online courses has definitely helped to reach millions of learners, but the educational 
effectiveness of online courses is a subject of debate. Learning must be personalized based on the learner’s goals 
and style and compared with “learner-like” learners (individualized and collaborative) as well as adaptive 
learning resources (organized and filtered), while considering motivation and engagement tools (Cheung, Lam, 
Szeto, & Yau, 2008). The goal of the adaptive presentation is to adapt the content to the user’s goals, knowledge, 
and other relevant information. The architecture for an Adaptive Hypermedia System adapts the content of a 
hypermedia page to the user’s goals, knowledge, preferences, and other user information for each individual user 
who is interacting with the system (Stern & Woolf, 2000). 

Another aspect of effective search and personalized results is consideration of the learner’s profile. All learners 
are unique; no two will achieve the same learning outcomes across a range of subject areas. Clear guidance can 
be provided on the diverse learning needs of each student by collecting and continuously updating metadata that 
is stored for learners in user profiles. Chan (2000) describes that implicit profile creation based on observations 
of users actions has been used in more recent projects and describes the types of information that is available. 
This model considers the frequency of visits to a page, the amount of time spent on each page, how recently a 
page was visited, and whether the page was bookmarked. Paireekreng & Wong (2010) observe that prior 
knowledge of each learner’s activity and an effective user profile is required for personalization. M.P. Cuellar, 
M. Delgado, and M.C. Pegalajar (2011) have considered social networks to be a type of Learning Management 
System (LMS). Social Network Analysis (SNA) is conducted for teachers, learners, learning resources and their 
interactions. Vassileva, J. (2008) emphasizes that the two main goals of the design of social learning 
environments should be making them learner-centered and making learning more gratifying. In recent research, 
association rule-mining algorithms have been used to solve the problem of web page recommendations. A web 
usage log is used in adaptive association rule-based web mining, which attempts to personalize the results. 

Research shows that effective learning requires the following: 

1. Learner centric adaptive learning by personalizing with relevant content based on the learner’s goals, 
style, habits and prior knowledge; 

2. Learner centric social learning based on the goals, learning style and behavioral patterns of similar 
learners; 

Current Open Content Learning systems include: OER Commons (Yoav Yair 2014, D’Antoni, S 2009), iseek.org 
(Bansal 2013), Project MERLOT (Malloy & Hanley 2001; Hanley 2015), OCW (Vahdati 2015) and mooc-list 
(Holotescu, Grosseck, Cretu & Naaji, 2014). Most of these systems are not personalized and do not provide 
adaptive content. Learners use these platforms as content viewers, and there is no engagement. They do not offer 
personalized content based on a learner’s goals and prior knowledge. To overcome these limitations, the 
proposed work is to develop an Open Content Repository by consuming the OER content and personalizing the 
learning experience based on the learner’s goals and activities and similar learners’ learning activities. 

Another aspect of effective search and personalized results is consideration of the learner’s profile. All learners 
are unique; no two will achieve the same learning outcomes across a range of subject areas. Clear guidance can 
be provided on the diverse learning needs of each student by collecting and continuously updating metadata that 
is stored for learners in user profiles. Chan (2000) describes that implicit profile creation based on observations 
of users actions has been used in more recent projects and describes the types of information that is available. 
This model considers the frequency of visits to a page, the amount of time spent on each page, how recently a 
page was visited, and whether the page was bookmarked. The user’s learning behavior is used to create user 
profiles in several systems. Paireekreng & Wong (2010) observe that prior knowledge of each learner’s activity 
and an effective user profile is required for personalization. Open pedagogy could be considered to be a blend of 
personalized adaptive design, algorithms and technologies, and networking among learners, which makes the 
learning process effective and engaging. 
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OPEN PEDAGOGY AND LEARNER-CENTERED LEARNING 

Some early MOOC experiments were based on a pedagogy of connectivist learning (Milligan, Littlejohn, & 
Margaryan, 2013), which connects many people in a loose online network that enables them to share their ideas 
and learn together. While this approach harnesses the power of many voices and technologies, it is difficult to 
manage at a large scale and requires learners to know how to navigate the web resources and engage with their 
peers (de Waard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez, & Gallagher, 2011). So which pedagogies 
actually improve with scale? Some effective methods of teaching, such as personal tutoring, cannot scale up to 
thousands of learners without enormous costs, even though researchers in artificial intelligence have been 
attempting for many years to develop computer-based tutors. In contrast, methods of direct instruction scale well 
- a good educational television program can inform a hundred people, or a million - but they are not very 
effective at engaging people in active and reflective learning. There is a general theory of scale that can be 
applied to education. The Network Effect proposes that the value of a networked product or service increases 
with the number of people who use it (Sharpies, Adams, Ferguson, Gaved, Me Andrew, Rienties, Weller & 
Whitelock, 2014). For example, a telephone system becomes more valuable when we connect millions or 
billions of phone users worldwide. The worldwide web benefits from interconnecting millions of people through 
their computers. But people are not solely points in a network; we have knowledge and perspectives to share. 
Thus, the Social Learning Effect can be stated as such: the value of a networked learning system increases as it 
enables people to learn easily and successfully from each other. Another difficulty experienced by many who 
have participated in connectivist MOOCs (Milligan, Littlejohn, & Margaryan, 2013) is the feeling of being dost 
in hyperspace,’ of having too many options and possibilities and not knowing where they are in a learning 
activity, who to engage with, and where to go next. 

Most existing e-learning platforms and tools focus on technology without rigorous investigation of the 
pedagogical issues or quality control of the e-leaming material. The motivation to learn and engage with the e- 
Learning solution is key to its effectiveness, especially when the effectiveness is defined as the time spent using 
the product: ‘Results suggest the importance of motivation to learn and workload in determining aggregate time 
spent in e-learning courses’ (Brown, 2005). Open pedagogy could be considered to be a blend of personalized 
adaptive design, algorithms and technologies, and networking among learners, which makes the learning process 
effective and engaging. 

OPEN CONTENT SOCIAL (OCSL) SYSTEM 

This section summarizes the general overall system architecture and design of OCSL before discussing the 
individual modules in detail. OCSL is a personalized learning system represented in figure 1 uses complex 
algorithms to automatically learn a learner’s interests with respect to learning activities. It then makes highly 
personalized content recommendations based on the goals, past activity and similar learners’ activities. 


Lea rnetv,Gentered Ex periencej 



Figure 1. Overview of the Learner-Centered Learning Experience leveraging Open Content. 

Research shows that most of the Open Content learning platforms currently use standard search techniques by 
combining conventional information retrieval techniques that are based on page content, such as word vector 
space (Salton, & McGill, 1983), with link analysis techniques based on the hypertext structure of the Web, such 
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as PageRank (Brin & Page, 1998) and HITS (Devi, Gupta, & Dixit, 2014). The PageRank algorithm (Brin & 
Page, 1998) attempts to provide an objective estimate of the Web page importance. However, the importance of 
the Web pages is subjective for different users. The true relevancy of a page depends on the interests, goals and 
existing knowledge of the individual users; a global ranking of a Web page might not necessarily capture the 
importance of a page for a given individual user. OCSL expands the scope of the search to generate more 
personalized results and greater learning engagement using the following two modules: 

A. Offline Process: 

1. The content manager reads the content (Crawling, API calls, Streaming API). 

2. The content classification engine analyzes the content. 

3. The system sends 20% of the content to the Natural Language Processing NLP API. 

4. After categorization, the content is verified by Amazon Mechanical Turk through APIs. 

5. The remaining 80% of the content is classified using the Naive Bayes classifier (Patil & Pawar 2012) 
algorithm. 

6. Once the content is classified with attributes (meta-data), it is loaded into the content index. 

The content index indexes the attributes and stores it inside the Apache Solr container. This content index is 
updated periodically through an offline process. 

2. Online Process: 

1. The learner inputs his/her goals, learning style, and relevant content. 

2. The pedagogy engine formulates the query to retrieve content in three ways, depending on the 
historical information and the learner’s goals: 

a. Conventional search using an inverted index and page ranking algorithm. 

b. Improved results based on the Content Hierarchy and Learner attribute-based Matching 
(CHLAM) of the OCSL system. 

c. Superior results based on CHLAM and Similar Learners Attribute-based Matching (CHSLAM) 
of the OCSL system. 

3. Filter the content results. 

4. Implicitly capture the learner’s activity and use it as a feedback loop to apply to the learner’s 
profile attributes. 

Each module performs its defined function and exchanges information with other modules, as shown in figure 2. 
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Figure 2. System Architecture of the OCSL Work 


The role of content discovery is to crawl open content from the Internet, i.e.„ the World Wide Web and social 
media, and to locate content to present to the user. The content manager is configured to collect content from 
three sources: 1. Crawling OER content sites 2. Streaming API against social media platforms 3. API calls 
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against learning platforms such as MERLOT (Hanley, 2015), OER Commons, Gooru learning. 

Content clustering entails grouping similar uncategorized documents together based on similarity measures. 
Content classification categorizes and organizes content by combining multiple methods of context-sensitive 
analysis. The clustering engine consumes content from multiple sources (Nutch Crawler, Federated API search, 
and Streaming API for social media feeds) and performs the following steps: 

1. Alchemy’s machine learning APIs (Quercia, Askham, & Crowcroft, 2012) are used for categorizing the 
content. OCSL uses the Taxonomy API to perform classification. The Entity API calls fetch the desired Internet 
web page, normalizes it, and extracst named entities, topics, and other content. 

a. http://www.alchemyapi.com/api/taxonomy_calls/urls.html 

b. http://www.alchemyapi.com/api/entity/urls.html#rurl 

Using the Taxonomy and Entity API, content metadata is updated in the Solr content repository. 

2. As recommended by Wang, Kraska, Franklin, & Feng (2012), OCSL leveraged a hybrid human-machine 
approach in which machines are used to perform an initial, coarse pass over all of the data, and people are used 
to verify only the most likely matching pairs. OCSL integrates with the Amazon Mechanical Turk API to verify 
the classified content. 

3. Using the Apache Mahout framework and Naive Bayes classifier algorithm (Patil & Pawar 2012), OCSL 
automatically classifies documents using a training set developed from the previous two steps. The training set 
includes documents that are already associated with a category. Using this set, the classifier determines, for each 
word, the probability that it reflects a document that belongs to each of the considered categories. To compute 
the probability that a document belongs to a category, the classifier multiplies together the individual 
probabilities of having each of its words in this category. The category that has the highest probability is the 
category that the document is most likely to belong to. 

4. OSCL updates the content index engine with all of the taxonomy attributes (URL, content category, content 
sub category, content type, last modified, and many more). 

The Dynamic Query Formulator is the core component of the OCSL system design. Most conventional search 
engines function with a search query that is limited and not as good as searching by phrases. The pedagogical 
engine uses a dynamic query formulator algorithm that was developed through this research to navigate a 
learner’s learning experience by analyzing his/her user interactions and prior learning knowledge on any given 
topic. The OCSL pedagogical engine also dynamically generates a query based on similar learners’ learning 
experiences. 

Learner Attribute-based Matching (LAM) enhances the conventional search experience by building a user profile 
to provide more personalized search results based on learning style, type of content, recent activity, content 
categories, or other interests of the users. To build an intelligent pedagogical learning engine based on attributes, 
this system ensures that both users and documents are tagged with the same types of attributes. We are implicitly 
and explicitly collecting information from learners about their learning behaviors, learning goals, and other 
criteria. Basically, the pedagogy engine is responsible for figuring out both the most appropriate way to construct 
the queries and which data to use in them to optimize the relevancy of the learner’s learning experience. While a 
conventional search engine builds a sparse matrix of terms that are mapped to documents in the content index, 
OCSL enhances the design to map the user’s behavior to those documents. The Learner Attribute-based Search 
enables the system to classify users and content into a hierarchy that goes from more general to more specific 
categories, but it is further possible to query this hierarchy and apply a stronger relevancy weight to more 
specific matches: 

Learner Profile: { 

MostLikely Category: "engineering, computer science, artificialintelligence ”, 

2ndMostLikely Category: "engineering, computerscience. datastructures ”, 

3rdMostLikelyCategory:"engineering.mathematics.algebra”, ... } 

First, each category from a learner’s profile can be broken into three terms in the query, with each term 
corresponding to a level of specificity in the classification: 

(engineering, computer science, artificialintelligence vv. . engineering, computerscience, datastructures vv. 
engineering, mathematics, algebra). 

Second, each term is assigned a different query weight, with higher weights assigned to more specific terms. 
This arrangement serves the purpose of boosting the more specific (and presumably better) matches higher in the 
search results. Third, there are three distinct sets of queries, which correspond to the three potential 
classifications that are listed on a learner’s profile: 
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(engineering, computerscience. artificialintelligence, engineering, computerscience. datastructures, 
engineering, mathematics, algebra). 

The end result is that by using query weights on terms that combine a measure of their probability (most likely to 
least likely) and their specificity (most descriptive to least descriptive), a fuzzy query can be constructed to 
match documents that match any of the criteria; at the same time, it boosts documents to the top of the search 
results that match the best combinations of those attributes within the hierarchy. 

The query parameter also allows the author to weight the fields differently. This parameter can be used to 
make a query match in one field more significant than a query match in another field. 

qf = field? + field? + - + field? 

where qf is the Query Fields, and v is the weight for each field, based on the learner’s goals and interests as 
calculated and applied dynamically. In our approach, we personalize PageRank scores by assigning weights to 
the fields based on matched goals and activities based on the learner and similar learners. At the query time, the 
user’s profile matches with the corresponding personalized values. 

By mapping the learning behavior of users to documents, OCSL system is effectively creating links in the index 
between documents. Klasnja-Milicevic, Vesin, Ivanovic, & Budimac (2011) recommended that similar users 
learn similar content, which means that documents that are mapped to similar users are likely related. To make 
use of these relationships to recommend learning items to a new user, we find other similar users and 
recommend other items. OCSL provides a mechanism to form a social network among the learners who have 
similar learning interests, preferences, and learning experiences based on the data collected. A learning group in 
OCSL is a group of learners who share common learning goals and mutually recommend learning content that 
meet those goals. OCLS uses User-based Collaborative filtering and Item-based Collaborative filtering 
(Drachsler, Hummel & Koper, 2008) to filter the learning content and recommend learning activities in a 
pedagogically effective order. 

To evaluate our design, we conducted a Web crawl against Open Educational Resources (OER) and 
implemented a dynamic query formulator engine. We performed an experimental study that focused on Science, 
Technology, Engineering, Mathematics (STEM) engineering students. Our study explored the results of the 
following three algorithms, to validate the idea of effective learning by personalizing the content results. The 
study lasted for almost three months. Learners were grouped into 15 groups. 

1. Algorithm 1 - Basic search using inverted index and page ranking conventional algorithm 

2. Algorithm 2 - Search based on the Content Hierarchy and Learner Attribute-based Matching (CHLAM) of the 
OCSL system 

3. Algorithm 3 - Search based on CHLAM and Similar Learners Attribute-based Matching (CHSLAM) of the 
OCSL system 

We asked each learner to use our OCSL system after they entered their goals and profiles into our system. We 
did not provide any information about the main goal of the system. The learners were expected to use the 
platform and learn based on their choice of preferences. A results page was shown with the recommended 
content based on the three different types of algorithms mentioned above. Figure 3 is a screen shot of the OCSL 
system. 
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Figure 3. OCSL System screen shot 


TESTING APPROACH AND RESULTS 

Comparing search results and recommendation systems is difficult. The best way to experiment with different 
relevancy parameters is to run A/B experiments that randomly divide users into groups over the same time 
period, with each group interacting with a different algorithm. Another common method for measuring the 
relative performance of algorithms involves generating test data and performing comparative analysis using the 
generated log data (Khosla, & Bhojane, 2013). To experiment with learning activities in detail, behavioral 
patterns were extracted from the log files and user activity database table. 


There are two aspects of a search result set that determine the quality of the results, the precision and recall, as 
Powers and David (Powers & David, 2011) suggest. Precision is the fraction of the retrieved documents that are 
relevant. A precision of 1.0 means that every result that is returned by the search is relevant, but there could be 
other relevant documents that were not a part of the search result. 


|{relevant documents} n {retrieved documents]\ 

precision = 

Recall is the fraction of the relevant documents that are retrieved. A recall of 1.0 means that all of the relevant 
documents were retrieved by the search, irrespective of the irrelevant documents also included in the result set. 


|{retrieved documents }| 


recall — 


|[relevant documents } n {retrieved documents]\ 
|{relevant documents] \ 


If all of the documents are retrieved, then the recall is perfect but the precision may not be good. On the other 
hand, if the document set contains only a single relevant document and that relevant document is retrieved in the 
search, then the precision is perfect but again the result set may not be good. This relationship shows a trade-off 
between the precision and recall, in which they are inversely related. 

The F-score is a measure of a test's accuracy. It considers both the precision p and the recall r of the test to 
compute the score: 

precision ■ recall 

Ft = 2 ■ —- 

precision + recall 

In this approach, we can take previously saved user behavior data from log files and test how well each of the 
candidate algorithms predicts the results that were previously acted on by the users. In the case of OCSL, we 
take the list of search results for every search or recommendation run for the user and plot them in aggregate on a 
precision versus recall graph, showing whether the algorithm made the correct prediction based on the user’s 
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historical behavior. For example, the correct prediction might be defined in terms of which learning materials a 
user consumed, and thus, any query model that resulted in higher precision and recall for that learning content 
would be considered to be a better algorithm. 

We analyzed the system logs and calculated the Precision, Recall and F-Score based on the learner’s activity for 
each algorithm. In the following results table, each row indicates the aggregated result of a group of learners who 
interacted with the system. The Learning activity indicates the number of times each learner interacted with the 
system. The Total recommendations show the number of learning (retrieved) documents that were displayed to 
the learners, while the Total documents indicate the possible number of documents (relevant documents) that 
were related to the search. 


Table 1. Conventional search using an inverted index and page ranking algorithm 


Group # 

# of interactions 

# of recommendations 

Total no. of documents 

Precision 

Recall 

F-Score 


1 

12 

510 

17519 

0.0235 

0.0007 

0.001368981 


2 

7 

2939 

15090 

0.0024 

0.0006 

0.000927337 


3 

4 

722 

17307 

0.0055 

0.0002 

0.000462127 


4 

9 

560 

17469 

0.0161 

0.0005 

0.001029866 


5 

38 

103 

16993 

0.0367 

0.0024 

0.004462451 


6 

35 

146 

17883 

0.2397 

0.002 

0.003906686 


7 

99 

172 

17857 

0.5756 

0.0056 

0.011026955 


8 

32 

660 

17369 

0.0485 

0.0019 

0.00367795 


9 

4 

459 

17570 

0.0087 

0.0002 

0.000455224 


10 

24 

1609 

16420 

0.0149 

0.0016 

0.002918998 


11 

20 

830 

17199 

0.0241 

0.0012 

0.002323015 


12 

77 

876 

17153 

0.0879 

0.0047 

0.008937899 


13 

32 

137 

17892 

0.2336 

0.0018 

0.003570632 


14 

30 

168 

16344 

0.0178 

0.002 

0.003664306 


15 

51 

80 

17949 

0.6375 

0.0028 

0.005666667 


Table 2. Search based on the Content Hierarchical and Learner Attribute-based Matching (CHLAM) of OCSL 

Group # 

# of interactions 

i # of recommendations 

Total no. of documents 

Precision 

Recall 


F-Score 

1 

123 

810 

[7219 

0.1519 

0.0074 


0.014185215 

2 

160 

616 

17413 

0.2597 

0.0094 


0,018209754 

3 

140 

439 

[7590 

0.3189 

0.0081 


0.015792442 

4 

120 

218 

17811 

0.5505 

0.0068 


0.013384641 

5 

230 

443 

17586 

0*5192 

0,0132 


0.025819488 

6 

230 

266 

17763 

0.8647 

0.013 


0.025565498 

7 

211 

612 

17417 

0,3448 

0.0124 


0,023939188 

3 

227 

389 

17640 

0.5835 

0.013 


0.025409974 

9 

211 

411 

17618 

0.5134 

0,0121 


0,023669303 

10 

220 

409 

17620 

0.5379 

0.0126 


0.024663677 

11 

166 

260 

17769 

0.6385 

0,0094 


0,018511291 

12 

121 

120 

16829 

0.1008 

0.0077 


0,014277286 

13 

177 

934 

17095 

0.1895 

0.0308 


0,0204956 

14 

156 

303 

17726 

0.5149 

0.0089 


0.017447713 

15 

1 JO 

900 

[7129 

0.1222 

0.0067 


0.012761761 


Table 3. Search based on CHLAM and on Similar Learners Attribute-based Matching (CHSLAM) 


Group tt 

# of interactions 

# of recommendations 

Total no. of documents 

Precision 

Recall 

F-Score 

1 

298 

330 

[7699 

0.903 

0.0169 

0,033116631 

2 

199 

260 

[7769 

0.7654 

0,0112 

0.02215049 

3 

82 

76 

[7953 

1.0789 

0.0046 

0.009093429 

4 

120 

140 

17889 

0.8571 

0.0067 

0.01332667 

5 

310 

311 

17718 

0,9968 

0.0175 

0,034390947 

6 

215 

217 

17812 

0,9908 

0.0121 

0.023853109 

7 

120 

124 

17905 

0,9677 

0,0067 

0,01331484 

S 

307 

330 

17699 

0.9303 

0.0174 

0.034099745 

9 

Ml 

101 

17928 

1,099 

0.0062 

0.012306669 

10 

130 

150 

17879 

0.8667 

0.0073 

0,014437226 

11 

144 

166 

17863 

0.8675 

0.0081 

0.01599378 

12 

168 

172 

17857 

0*9767 

0.0094 

0,018640777 

13 

141 

146 

17883 

0,9658 

0.0079 

0,015645806 

14 

318 

320 

17709 

0.9938 

0.018 

0,035280413 

15 

1 19 

120 

[7909 

0,9917 

0.0066 

0,013201686 


The data in the table represents aggregate precision and recall calculations that are based on the learners in 15 
different groups. Table 3 shows that the learning groups that used OCSL with the CHSLAM algorithm had an 
effective learning experience by interacting with the system more than the user groups that used the OCSL with 
the conventional and CHLAM algorithms. The precision is calculated as (# correct matches) / (# total results 
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returned), and the recall is calculated as (# correct matches) / (# correct matches + # missed matches). Although 
the precision and recall are not perfectly negatively correlated, there is a natural tension between the two in such 
a way that improvements in one often lead to declines in the other. The data from the table can be easily turned 
into a graph. All three tables are generated as graphs in Figure 4, Figure 5, and Figure 6, which show that the 
CHSLAM algorithm of OCSL generates improved results. 



Figure 4. Precision values for Conventional, CHLAM and CHSLAM of OCSL algorithms 


iventional 
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0.0100 


Recall 
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Figure 5. Recall values for Conventional, CHLAM and CHSLAM of OCSL algorithms 
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Figure 6. F-Score values for Conventional, CHLAM and CHSLAM of OCSL algorithms 

The F-score shows an absolute score for an algorithm that strives for good balance between the precision and 
recall. Figure 6 shows that the learners engaged more successfully based on the CHSLAM algorithm compared 
to the CHLAM and conventional algorithms. The F-Score can be interpreted as a weighted average of the 
precision and recall, where an F-Score reaches its best value at 1 and worst at 0. The average F-Score value for 
conventional algorithm was 0.0034, and for CHLAM algorithm it was 0.0190 and for CHSLAM algorithm it was 
0.0203. Based on the tests, CHSLAM algorithm yielded better F-Score results. To obtain a subjective evaluation 
of the OCSL system, we organized a non-mandatory questionnaire that collected information on learners with 
respect to the main features of the system. More than 65% of the learners reported that the system recommended 
personalized results and was able to focus on the correct content. Overall, the system showed remarkable 
improvement in self-learning. The learners were able to focus more time on studying the correct content and less 
time on searching for the content. 

CONCLUSIONS 

We presented a design and implementation of an end-to-end implementation model and conducted several 
experiments to test our system. Our system starts with a clustering engine that processes the content from various 
OER sources to properly map it to the taxonomy we built to support STEM (science, technology, engineering, 
and mathematics) content. It then generates personalized search results based on the content hierarchy (e.g., 
content type, content category) and learner attributes (e.g., learning style, recent activity). We took the learner 
experience from the logs and database and plotted them in aggregate on a precision versus recall graph, which 
showed whether the algorithm made the correct prediction based on the learner’s historical behavior as well as 
similar learners’ learning behaviors. Here, the precision and recall are not perfectly negatively correlated; there is 
a natural tension between the two in such a way that improvements in one often lead to declines in the other. We 
found that a search that was based on the historical learning of learners and similar learners’ behaviors 
(CHSLAM of OCSL) yielded better F-Score results compared with the conventional search as well as a search 
based only on Content Hierarchical and Learning Attribute-based Learning (CHLAM). In the future, we plan to 
expand the system by creating peer groups with complex algorithms by leveraging similar learners’ data from 
OCSL. We will explore extending the personalized mechanism and pedagogical aspects of OCSL to increase the 
engagement of learners by having the influences and mentors interact with the peer group. 
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